Arabic Dialect Processing Tutorial

The existence of dialects for any language constitutes a challenge for NLP in general since it adds another set of variation dimensions from a known standard. The problem is particularly interesting and challenging in Arabic and its different dialects, where the diversion from the standard could, in some linguistic views, warrant a classification as different languages. This problem would not be as pronounced if Modern Standard Arabic (MSA) were the native language of some sub group of Arabic speakers, however it is not. Any realistic and practical approach to processing Arabic will have to account for dialectal usage since it is so pervasive. In this tutorial, we will attempt to highlight different dialectal phenomena, how they migrate from the standard and why they pose challenges to NLP. This area of research (dialects in general and Arabic dialects in particular) is gaining a lot of interest. For example, the DARPA-funded BOLT program starting this year will only consider dialectal varieties for its effort on Arabic. Furthermore, there was a workshop on dialect processing as part of EMNLP 2011. This tutorial has four different parts: First, we contextualize the question of Arabic dialects from a sociolinguistic and political perspective. Second, we present a discussion of issues in relevant to Arabic NLP; this includes generic issues common to MSA and dialects, and MSA specific issues. In the third part, we detail dialectal linguistic issues and contrast them to MSA issues. In the last part, we review the stateof-the-art in Arabic dialect processing covering several enabling technologies and applications, e.g., dialect identification, speech recognition, morphological processing (analysis, disambiguation, tokenization, POS tagging), parsing, and machine translation. Throughout the presentation we will make references to the different resources available and draw contrastive links with standard Arabic and English. Moreover, we will discuss annotation standards as exemplified in the Treebank. We will provide links to recent publications and available toolkits/resources for all four sections. This tutorial is designed for computer scientists and linguists alike. No knowledge of Arabic is required (though, we recommend taking a look at Nizar Habash's Arabic NLP tutorialhttp://www1.ccls.columbia.edu/~cadim/presentations.html which will be reviewed as part of the tutorial.)